Collaborators

Russ Thurow USDA Forest Service Rocky Mountain Research Station

Claire McGrath Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region,

Kevin See Biometrician, Biomark Inc, Boise, ID,

Goals

Salmon redd counts are widespread method to estimate the number of returning adult spawners. However, despite its prevalence in the Northwest, the reliability of redd counts is unknown. This work is focused on developing a statistical model to estimate the observer error in redd surveys, using a variety of covariates related to the habitat and the observer. We described three types of observer error:

  • omission rate, \(\omega\) (proprotion of redds available to be counted that were missed by the observer)
  • commission rate, \(\eta\) (rate of redds counted by an observer that were not actually redds)
  • net error, \(\gamma\) (ratio of observed redds to true redds). This was modeled using log(net error) as the reponse.

Methods

Possible covariates in each error model are shown in Table 1. To make comparisons with AICc, the random effects must be identical across all models. Therefore, we ensured that the random effect of year was added to any model that didn’t have it.

Table 1: Possible covariates included in each observer error model.

Type Covariate Air Ground
Random Reach X X
Random Surveyor X
Random Year X X
Fixed Alluvium X X
Fixed ANNDist_log X X
Fixed AveAge X X
Fixed AveBadCond X
Fixed AveCanopy X X
Fixed AveContrast X X
Fixed AveDepth X X
Fixed AveOverlap X X
Fixed AveSunny X
Fixed AveWidth X X
Fixed ExperienceCat X
Fixed I(AveDepth^2) X X
Fixed Lithology X X
Fixed LYabund X X
Fixed LYabund:PeakQ X X
Fixed OthrDens X X
Fixed PeakQ X X
Fixed redd_dens X X
Fixed Slope X X

All covariates were z-scored, and all models were fit using the glmer or lmer functions from the lme4 package (Bates et al. 2015) in R software (R Core Team 2019). The amount of variation explained by fixed and random effects was calculated using the methods of Nakagawa and Schielzeth (2013). Using estimated predictions of the rates for omission (\(\hat{\omega}\)), commission (\(\hat{\eta}\)) and net error (\(\hat{\gamma}\)), we predicted the number of actual redds by either dividing the observed counts, \(c\), by estimates of net error, or by multiplying the observed counts by 1 - estimated rate of commission, and then dividing by 1 - estimated rate of omission.

We performed a cross validation by dividing each survey type data into 10 training datasets where 20% of the data was withheld for testing, and then fitting the naive and best AICc model formulations to the remaining data, and then using those fits to predict the error rates and true number of redds for each survey in the year that had been withheld.

\[ \begin{aligned} redds_{ne} &= \frac{c}{\hat{\gamma}} \\ redds_{om} &= c * \frac{1 - \hat{\eta}}{1 - \hat{\omega}} \end{aligned} \]

The observed error rates are showin in Figure 1.

Figure 1: Observed error rates.

Figure 1: Observed error rates.

Results

Model Coefficients

The model coefficients of the full, best (by AICc) and model averaged models are shown in Table 2.

Table 2: Estimated coefficients for various observer error models.

Survey Resp Covariate avg best full
Ground Com (Intercept) -1.917 -1.917 -1.917
Ground Com AlluviumN 0.721 0.721 0.721
Ground Com ANNDist_log 0.386 0.386 0.386
Ground Com AveAge -0.034 -0.034 -0.034
Ground Com AveCanopy 0.238 0.238 0.238
Ground Com AveContrast 0.013 0.013 0.013
Ground Com AveDepth 0.315 0.315 0.315
Ground Com AveOverlap 0.077 0.077 0.077
Ground Com AveWidth -0.043 -0.043 -0.043
Ground Com ExperienceCat.L -0.189 -0.189 -0.189
Ground Com ExperienceCat.Q 0.327 0.327 0.327
Ground Com I(AveDepth^2) 0.008 0.008 0.008
Ground Com Lithology2 1.012 1.012 1.012
Ground Com Lithology3 -0.286 -0.286 -0.286
Ground Com Lithology5 1.165 1.165 1.165
Ground Com LYabund 0.325 0.325 0.325
Ground Com LYabund:PeakQ 0.190 0.190 0.190
Ground Com OthrDens 0.212 0.212 0.212
Ground Com PeakQ -0.084 -0.084 -0.084
Ground Com redd_dens 0.011 0.011 0.011
Ground Com Slope -0.142 -0.142 -0.142
Ground Net (Intercept) -0.390 -0.390 -0.824
Ground Net AlluviumN 0.272 - 0.272
Ground Net ANNDist_log 0.260 0.258 0.297
Ground Net AveAge -0.055 - -0.055
Ground Net AveCanopy 0.145 - 0.145
Ground Net AveContrast 0.022 - 0.022
Ground Net AveDepth 0.156 - 0.156
Ground Net AveOverlap -0.004 - 0.000
Ground Net AveWidth -0.034 - -0.034
Ground Net ExperienceCat.L 0.270 - 0.270
Ground Net ExperienceCat.Q -0.156 - -0.156
Ground Net I(AveDepth^2) -0.029 - -0.029
Ground Net Lithology2 0.673 - 0.673
Ground Net Lithology3 -0.125 - -0.125
Ground Net Lithology5 0.684 - 0.684
Ground Net LYabund 0.301 - 0.301
Ground Net LYabund:PeakQ 0.240 - 0.240
Ground Net OthrDens 0.030 - 0.030
Ground Net PeakQ 0.018 - 0.018
Ground Net redd_dens 0.049 - 0.007
Ground Net Slope -0.116 - -0.116
Ground Omi (Intercept) 0.175 0.175 0.175
Ground Omi AlluviumN 0.212 0.212 0.212
Ground Omi ANNDist_log -0.082 -0.082 -0.082
Ground Omi AveAge 0.137 0.137 0.137
Ground Omi AveCanopy -0.075 -0.075 -0.075
Ground Omi AveContrast -0.037 -0.037 -0.037
Ground Omi AveDepth 0.098 0.098 0.098
Ground Omi AveOverlap 0.087 0.087 0.087
Ground Omi AveWidth -0.098 -0.098 -0.098
Ground Omi ExperienceCat.L -0.777 -0.777 -0.777
Ground Omi ExperienceCat.Q 0.663 0.663 0.663
Ground Omi I(AveDepth^2) -0.031 -0.031 -0.031
Ground Omi Lithology2 -0.593 -0.593 -0.593
Ground Omi Lithology3 -0.171 -0.171 -0.171
Ground Omi Lithology5 -0.796 -0.796 -0.796
Ground Omi LYabund -0.535 -0.535 -0.535
Ground Omi LYabund:PeakQ -0.692 -0.692 -0.692
Ground Omi OthrDens -0.010 -0.010 -0.010
Ground Omi PeakQ -0.201 -0.201 -0.201
Ground Omi redd_dens 0.201 0.201 0.201
Ground Omi Slope 0.300 0.300 0.300
Survey Resp Covariate avg best full
Air Com (Intercept) -1.341 -1.342 -3.323
Air Com AlluviumN 0.927 - 0.926
Air Com ANNDist_log 0.382 0.290 0.302
Air Com AveAge -0.156 - -0.118
Air Com AveBadCond 0.022 - 0.022
Air Com AveCanopy 0.386 - 0.387
Air Com AveContrast 0.007 - 0.052
Air Com AveDepth 0.047 - 0.047
Air Com AveOverlap -0.136 -0.136 -0.174
Air Com AveSunny 0.176 - 0.156
Air Com AveWidth 0.345 - 0.345
Air Com I(AveDepth^2) -0.001 - -0.001
Air Com Lithology2 2.083 - 2.083
Air Com Lithology3 0.014 - 0.013
Air Com Lithology5 2.289 - 2.289
Air Com LYabund 0.785 - 0.786
Air Com LYabund:PeakQ 0.736 - 0.737
Air Com OthrDens 0.051 - 0.051
Air Com PeakQ -0.128 - -0.127
Air Com redd_dens -0.204 -0.198 -0.175
Air Com Slope -0.362 - -0.362
Air Net (Intercept) -0.402 -0.401 -1.125
Air Net AlluviumN 0.321 - 0.330
Air Net ANNDist_log 0.283 0.303 0.233
Air Net AveAge -0.179 - -0.182
Air Net AveBadCond 0.001 - 0.027
Air Net AveCanopy -0.077 - 0.179
Air Net AveContrast 0.078 - 0.064
Air Net AveDepth 0.073 - 0.068
Air Net AveOverlap -0.102 - -0.116
Air Net AveSunny 0.229 - 0.058
Air Net AveWidth 0.119 - 0.125
Air Net I(AveDepth^2) -0.004 - -0.002
Air Net Lithology2 0.872 - 0.892
Air Net Lithology3 0.047 - 0.000
Air Net Lithology5 1.057 - 1.070
Air Net LYabund 0.303 - 0.441
Air Net LYabund:PeakQ 0.283 - 0.436
Air Net OthrDens -0.167 - -0.050
Air Net PeakQ -0.132 - -0.099
Air Net redd_dens 0.010 - -0.020
Air Net Slope -0.186 - -0.209
Air Omi (Intercept) -0.565 -0.565 0.342
Air Omi AlluviumN 0.004 - 0.004
Air Omi ANNDist_log -0.038 - -0.190
Air Omi AveAge 0.507 0.507 0.574
Air Omi AveBadCond -0.025 - -0.025
Air Omi AveCanopy -0.084 - -0.084
Air Omi AveContrast 0.014 0.014 -0.008
Air Omi AveDepth 0.303 - 0.303
Air Omi AveOverlap 0.287 0.287 0.190
Air Omi AveSunny -0.032 - -0.031
Air Omi AveWidth -0.193 - -0.193
Air Omi I(AveDepth^2) -0.060 - -0.060
Air Omi Lithology2 -1.088 - -1.088
Air Omi Lithology3 -0.491 - -0.491
Air Omi Lithology5 -1.012 - -1.012
Air Omi LYabund -0.451 - -0.452
Air Omi LYabund:PeakQ -0.724 - -0.725
Air Omi OthrDens 0.103 - 0.103
Air Omi PeakQ 0.159 - 0.160
Air Omi redd_dens 0.152 - -0.095
Air Omi Slope 0.360 - 0.360

Ground Surveys

The relative importance of each covariate in each model is shown in Figure 2, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 3. Observed versus predicted rate plots are shown in Figures 4, 6 and 8.

Figure 2: Relative importance of each covariate in ground-based observer error models

Figure 2: Relative importance of each covariate in ground-based observer error models

Figure 3: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 3: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Omission

Figure 4: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 4: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 5: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 5: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Commission

Figure 6: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 6: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 7: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 7: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Net Error

Figure 8: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 8: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 9: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 9: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Leave-One-Out Cross Validation

Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects).

Redd Estimates

For ground-based surveys, both methods provided fairly unbiased estimates of the true number of redds (Figure 10), although the omission/commision models had slightly higher absolute and relative bias (Table 3).

Figure 10: Boxplots of absolute and relative bias for each type of predictive model.

Figure 10: Boxplots of absolute and relative bias for each type of predictive model.

Table 3: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.

Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 36 38 6.0 -0.3 -1.5 24.0
Best Omis / Comm Error 36 38 4.3 -0.9 -4.8 16.1
Naive Net Error 36 38 5.2 -0.9 -2.7 23.8
Naive Omis / Comm Error 36 38 4.1 -0.2 -0.6 18.2
Observed 36 38 - -2.0 -8.0 18.9
Figure 11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Air Surveys

The relative importance of each covariate in each model is shown in Figure 12, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 13. Observed versus predicted rate plots are shown in Figures 14, 16 and 18.

Figure 12: Relative importance of each covariate in ground-based observer error models

Figure 12: Relative importance of each covariate in ground-based observer error models

Figure 13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Omission

Figure 14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Commission

Figure 16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Net Error

Figure 18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Leave-One-Out Cross Validation

Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects).

Redd Estimates

For air-based surveys, both methods provided estimates of the true number of redds that were biased high (Figure 20). However, the net error models had lower absolute and relative bias, as well as a smaller root squared mean error (RMSE) (Table 4).

Figure 20: Boxplots of absolute and relative bias for each type of predictive model.

Figure 20: Boxplots of absolute and relative bias for each type of predictive model.

Table 4: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.

Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 32 38 9.7 0.5 1.9 23.8
Best Omis / Comm Error 32 38 6.1 0.3 1.0 21.6
Naive Net Error 32 38 6.6 0.2 0.4 27.0
Naive Omis / Comm Error 32 38 7.8 -1.2 -3.1 27.7
Observed 32 38 - -6.0 -18.2 28.7
Figure 21: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 21: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Discussion

References

Bates, D., Mächler, M., Bolker, B., and Walker, S. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1): 1–48.

Nakagawa, S., and Schielzeth, H. 2013. A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4(2): 133–142. Wiley Online Library.

R Core Team. 2019. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.

  1. Biometrician, Biomark, Inc., ↩︎

  2. Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region, ↩︎

  3. USDA Forest Service Rocky Mountain Research Station↩︎